Feature selection - one form of dimension reduction

Even if \(X\) full rank, we might want eliminate some predictors to decrease model variance.

• near - zero variance predictors

• highly correlated predictors (with each other)

• predictors uncorrelated with the response

• Forward/backward stepwise selection
• Lasso regression
• …

Curse of dimensionality - simple analytical example

Suppose KNN with \(p\) predictors \(\sim U(0,1)\)

• volume of hypercube with sides of length \(d\) is \(d^p\).

• To capture \(r\) proportion of the data, we want \(d^p = r\), so \(d = r^{1/p}\)

Features

• predictors or variables used in a model

• transformed and combined versions of the original predictors

Feature engineering is the art of creating useful features; it may increase or decrease the number of predictors.

Dimension reduction

Reducing the number of features/dimension of feature space

• Feature selection - select a subset of the original predictors

• Feature extraction - create new predictors from the original ones

  • PCA, UMAP, tSNE, neural networks/deep learning

Principal Components Analysis (PCA)

Create features - components from linear combinations of the predictors

• The first component is in the direction of maximum variance of the data*.

• Second component in direction of max variance orthogonal (and this uncorrelated) to the first, and so on.

PCA - the math

Predictors \(X_1, X_2, \dots, X_p\); form linear combinations

\[ Z_m = \sum_{j=1}^p \phi_{jm} X_j, \qquad m = 1, 2, \dots, p \]

In matrix form,

\[ Z = X \mathbf{U} \]

Now with data

\[ \mathbf{Z} = \mathbf{XU} \]

Principal components and eigenvectors

First scale or standardize the data matrix.

The principal components (columns of \(\mathbf{U}\)) are the eigenvectors of \(\mathbf{X}^T \mathbf{X}\).

(Equivalently, the eigenvectors of the correlation matrix - and dont need to standardize.)

Theorem

• Eigenvectors of \(\mathbf{X}^T \mathbf{X}\) are orthogonal

• Let \[ \lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_p > 0 \] with eigenvectors \(\mathbf{v}_1, \dots, \mathbf{v}_p\)

  • \(\lambda_i\) is variance in direction \(\mathbf{v}_i\)
  • \(\mathbf{v}_1\) maximizes variance
  • \(\mathbf{v}_2\) maximizes variance subject to orthogonality with \(\mathbf{v}_1\)
  • and so on

Principal components regression (PCR) in practice

1. Center and scale the data

2. Form \(\mathbf{U}\), the eigenvectors of the centered and scaled \(\mathbf{X}^T \mathbf{X}\), ordered by decreasing eigenvalues

3. Compute \(\mathbf{Z} = \mathbf{XU}\)

4. Select the number of components based on explained variance (or cross-validation)

5. Fit a linear regression using the first \(m\) components

Example: meatspec data

Analytical chemistry - predict fat content of meat samples from 100 channel infrared absorbance spectra.

PCA components cumulative variances

 [1] 0.98626 0.99596 0.99875 0.99990 0.99996 0.99999 0.99999 1.00000 1.00000
[10] 1.00000

R squared vs RMSE

Observed versus predicted values for models that are optimized using the RMSE compared to the coefficient of determination

Source: Tidy Modeling with R, Max Kuhn and Julia Silge, 2023

Example: wine data (UCI ML Repository)

Chemical analysis of wines grown in the same region in Italy but derived from three different cultivars. The analysis determined the quantities of 13 constituents found in each of the **three types* of wines.

Total and explained variance of the PCA components

Multinomial regression on the PCA features

We get perfect score with just 5 components, and 98% with only 3 components.

Loadings and scores

 [1] 0.361988 0.554063 0.665300 0.735990 0.801623 0.850981 0.893368 0.920175
 [9] 0.942397 0.961697 0.979066 0.992048 1.000000

Warning

These are “unscaled” loadings and scores. The prcomp function in R uses scaling by default.

Visualizing the loadings

First four PCAcomponents

Example: USArrests dataset¹

	Murder	Assault	UrbanPop	Rape
Alabama	13.2	236	58	21.2
Alaska	10.0	263	48	44.5
Arizona	8.1	294	80	31.0
Arkansas	8.8	190	50	19.5
California	9.0	276	91	40.6
Colorado	7.9	204	78	38.7

Loadings and scores

PCA loadings; the \(\mathbf{U}\) values

	PC1	PC2	PC3	PC4
Murder	-0.536	-0.418	0.341	0.649
Assault	-0.583	-0.188	0.268	-0.743
UrbanPop	-0.278	0.873	0.378	0.134
Rape	-0.543	0.167	-0.818	0.089

Cumulative normalized variances

[1] 0.620060 0.867502 0.956642 1.000000

PCA scores, \(\mathbf{Z} = \mathbf{XU}\)

	PC1	PC2	PC3	PC4
Alabama	-0.975660	-1.122001	0.439804	0.154697
Alaska	-1.930538	-1.062427	-2.019500	-0.434175
Arizona	-1.745443	0.738460	-0.054230	-0.826264
Arkansas	0.139999	-1.108542	-0.113422	-0.180974
California	-2.498613	1.527427	-0.592541	-0.338559
Colorado	-1.499341	0.977630	-1.084002	0.001450
Connecticut	1.344992	1.077984	0.636793	-0.117279
Delaware	-0.047230	0.322089	0.711410	-0.873113
Florida	-2.982760	-0.038834	0.571032	-0.095317
Georgia	-1.622807	-1.266088	0.339018	1.065974

Visualizing the feature contributions in the components

Biplots

We can visualize the loadings and the scores together with a biplot.

Non-linear dimension reduction

UMAP, t-SNE, autoencoders (neural networks)

UMAP: penguins revisited

We use just the four numerical predictors

species	island	bill_length_mm	bill_depth_mm	flipper_length_mm	body_mass_g	sex	year
Adelie	Torgersen	39.1	18.7	181	3750	male	2007
Adelie	Torgersen	39.5	17.4	186	3800	female	2007
Adelie	Torgersen	40.3	18.0	195	3250	female	2007
Adelie	Torgersen	NA	NA	NA	NA	NA	2007
Adelie	Torgersen	36.7	19.3	193	3450	female	2007
Adelie	Torgersen	39.3	20.6	190	3650	male	2007

PCA vs UMAP on penguins

Penguins pairplot

UMAP may be overkill here

UMAP

mnist dataset² - images of handwritten digits (0-9). 784 features (28 x 28 pixels)

PCA vs UMAP on mnist

Neural Networks for Dimension Reduction

Source: Gerhard Paab blog

What we covered

• Why dimension reduction matters

  • High-dimensional predictors and rank deficiency
  • Variance, multicollinearity, and computational cost
  • The curse of dimensionality (analytical + simulation intuition)

• Two broad strategies

  • Feature selection:
  • Feature extraction:

• Principal Components Analysis (PCA)

• Class experiments/examples

  • meatspec: PCR with many correlated predictors
  • wine: classification with few principal components
  • USArrests: interpreting loadings, scores, and biplots
  • penguins: PCA vs UMAP for visualization

• Nonlinear dimension reduction

  • UMAP, neural networks